Linking a physical item to a virtual item

ABSTRACT

Disclosed are systems and methods for mapping a virtual shopper to a physical shopper. The systems and methods may include receiving, at a backend system, unidentified customer data including information about an unidentified customer. Customer identity data including identifying information about the customer may be received from the customer. An authentication token may be generated that links the unidentified customer data to the customer identity data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/174,912, filed Oct. 30, 2018, which applications and publications areincorporated herein by reference in their entirety.

BACKGROUND

Computer vision and image processing techniques have advanced in recentyears. However, one difficult and unsolved processing task includesknowing when to associate one tracked item with another tracked item.This has been particularly difficult in frictionless store systems wheredecisions have to be made from the images as to whether an item capturedin a first part of a store is the same item captured in a second part ofthe store.

SUMMARY

Disclosed are systems and methods for mapping a virtual shopper to aphysical shopper. The systems and methods may include receiving, at abackend system, unidentified customer data including information aboutan unidentified customer. Customer identity data including identifyinginformation about the customer may be received from the customer. Anauthentication token may be generated that links the unidentifiedcustomer data to the customer identity data.

BRIEF DESCRIPTION OF THE FIGURES

The above-mentioned and other features and advantages of this invention,and the manner of attaining them, will become more apparent and theinvention itself will be better understood by reference to the followingdescription of embodiments of the invention taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 shows an example operating environment consistent with thisdisclosure.

FIG. 2 shows an example schematic of a server consistent with thisdisclosure.

FIG. 3 shows an example method consistent with this disclosure.

Corresponding reference characters indicate corresponding partsthroughout the several views. The exemplifications set out hereinillustrate exemplary embodiments of the invention, and suchexemplifications are not to be construed as limiting the scope of theinvention any manner.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar elements.While embodiments and examples are described, modifications,adaptations, and other implementations are possible. For example,substitutions, additions, or modifications may be made to the elementsand stages illustrated in the drawings, and the systems and methodsdescribed herein may be modified by substituting, reordering, or addingstages to the disclosed methods or elements to the disclosed systems.Accordingly, the following detailed description does not limit thisdisclosure. Instead, the proper scope of any invention disclosed hereinis defined by the appended claims.

When a shopper enters a frictionless checkout store they may be asked tocheck-in in a fixed physical location. The systems and methods disclosedherein allow for a variety of methods for an authentication token thatmaps the physical shopper to a virtual identity.

When a customer is in the frictionless checkout store and they pick upgoods, a virtual identity may be used to track and charge them. Thesystems and methods disclosed herein may allow for the mapping aphysical shopper in the frictionless checkout store's check-in to avirtual identifier. The virtual identifier may then be used to track thephysical shopper throughout the store and account for products thecustomer wishes to purchase.

To map the physical shopper to the virtual identity a designatedcheck-in zone may be monitored by sensors such as cameras, scales,BLUETOOTH® or other beacons, etc. The shopper may check-in or bechecked-in automatically to link the shopper to their virtual identity.Example ways to check in may include the use of quick response (QR)codes, biometrics, use of mobile devices, etc. For example, a customermay scan a QR code to provide a virtual identity. Also, the customer mayregister their facial image and a biometric face scan may be used toprovide their virtual identity. Furthermore, the customer may downloadan app which can allow for communications with a backend system toprovide their virtual identity.

The virtual identity may be associated with a register account such thatcredit card information, shopping preferences, etc. may be provided. Thecredit card information shopping preferences, etc. may be saved in auser profile. The various information may be collected and/or updated asthe customer shops. Once the customer leaves the frictionless checkoutstore with goods, he or she may be charged using the credit cardinformation saved in the user profile.

Without the systems and methods disclosed herein, the would be no way toidentify a customer and correctly link them to a virtual shopper via anauthentication token. Without the ability to link a customer to avirtual shopper, there would be no way to charge the correct accountwhen the customer exits the frictionless checkout store.

FIG. 1 shows an example operating environment 100 consistent with thisdisclosure. As shown in FIG. 1, operating environment 100 may include astore 102. Within store 102 may be a plurality of sensors 104(individually sensors 104A through 104N) and a server 106. Sensors 104may be in communications with server 106 via a network 108. Also, store102 may also include a check-in area 110. While FIG. 1 shows threesensors located at check-in area 110, any number of sensors may be usedto capture unidentified customer data about a customer 112.

During use, customer 112 may enter store 102 and pass into check-in area110. Upon entering store 102, sensors 104 may collect data aboutcustomer 112. The data, sometimes referred to as unidentified customerdata because the data is associated with an unidentified customer, maybe transmitted from sensors 104 to server 106 via network 108.

Collecting the unidentified customer data may include collecting avariety of information about customer 112. For example, one or more ofsensors 104 may be a camera and may capture images of customer 112. Oneor more of sensors 104 may be a scale that captures a weight of customer112. One or more of sensors 104 may be a beacon, such as a BLUETOOTH®beacon that captures information from a mobile device of customer 112.In addition, the unidentified customer data may be collected by morethan one sensor or method. For instance, the weight of customer 112 maybe captured in addition to images of customer 112.

Depending on how the unidentified customer data is collected additionalprocessing may be needed. For instance, if images of customer 112 arecaptured, image analysis may be used to extract information aboutcustomer 112. For example, the images may be cropped to removeextraneous imagery. For instance, server 106 may utilize images of knownbackground elements to filter the background elements from any imagescaptured by sensors 104.

Upon being received at server 106, the images may be processed by anencoder that may extract attributes about customer 112. Extractingattributes about customer 112 may include determining color, textwritten on a shirt, a hat, or other clothing worn by customer 112, aheight of customer 112, approximate weight of customer 112, color ofclothing worn by customer 112, etc.

The number of attributes may be preset or may depend upon the imagescollected. For example, the number of attributes may be a set numbersuch as N. In another example, the number of attributes may bedetermined by the number of unique features that can be identified fromthe images. For instance, using a background reference, a height ofcustomer 112 as well as a color of his or her shirt, pants, shoes(including logos upon the shirt, pants, and shoes) may be uniquelyidentified from the images.

The attributes extracted from the images along with any other datacollected in check-in area 110 may be assigned an initial identificationvalue and saved within an k-nearest neighbor (k-NN) database. Asdisclosed herein, the k-NN database may be saved on a memory of server106 or may be stored offsite.

While in check-in area 110 customer 112 may provide his or her customeridentity data. The customer identity data may be provided by thecustomer in a multitude of ways. For instance, upon entering check-inarea 110 one of sensors 104, which may be a camera, may capture an imageof customer 112. The image may be used by server 106 to perform a facialrecognition process to identify customer 112. In another instance,customer 112 may approach a kiosk and provide identifying informationsuch as a username and password or provide a QR code that can be scannedat the kiosk or with one of sensors 104. In yet another instance, uponentering check-in area 110 a mobile device of customer 112, running anapp provided by store 102, may push a username and password or otheridentifying credentials to server 106 via network 108. In addition, thecustomer identity data may be provided in more than one manner. Forinstance, facial recognition may be used in conjunction credentialsbeing pushed from a mobile device of customer 112.

With the unidentified customer data and the customer identity data,server 106 may generate an authentication token that links customeridentity data to the unidentified customer data. The authenticationtoken can then be associated with a virtual shopper. The virtual shoppercan then be used to track customer 112 as customer 112 walks throughstore 112.

For example, as customer 112 walks through store 102, additional imagesof customer 112, or an item customer 112 handles or looks at, may becaptured by sensors 104. These new images may be received by server 106and features from the new images extracted as described herein. Theextracted attributes may be stored in a feature vector and assigned atemporary identification value.

The feature vector and the temporary identification value may besubmitted to server 106 as a query. Server 106 may then determine whichof the feature vectors stored in a k-NN database the feature vectorassociated with the temporary identification value is closest to andthus determine that the customer in the second set of images is customer112. Stated another way, server 106 may determine the identity ofcustomer 112 as he or she moves throughout store 102 by consistentlymapping attributes from a plurality of images collected by sensors 104as customer 112 walks through store 102 using the attributes and aninitial set of attributes stored in the k-NN database along with theauthentication token and the customer identity data.

Using server 106 in conjunction with sensors 104 customer 112 can betracked throughout store 102. The departments visited and the itemscustomer 112 looks at and places in a shopping cart may be tracked andstored in a user profile. The user profile may be used to target ads,coupons, etc. to customer 112 while in store 102.

When customer 112 exits store 102 customer data, authentication token,etc. may be flushed from the k-NN database. The flushing of customer112's information may allow for faster processing of other featurevectors as they cannot be a match to customer 112's attributes if he orshe has left store 102.

In addition, when customer 112 exits store 102, any items customer 112may have in a shopping cart or that may have been scanned with customer112's mobile device may be charged to an account or credit cardbelonging to customer 112 and stored with a user profile of customer112.

FIG. 2 shows an example schematic of server 106. As shown in FIG. 2,server 106 may include a processing unit 202 and a memory 204. Thememory 204 may include a software module 206, unidentified customer data208, and customer identity data 210. While executing on processing unit202, the software module 206 may perform processes for linking aphysical shopper to a virtual shopper as disclosed herein, including,for example, one or more stages included in a method 300 described belowwith respect to FIG. 3. Server 106 may also include a user interface212, a communications port 214, and an input/output (I/O) device 216.

As disclosed herein, unidentified customer data 208 may include datathat is collected from sensors, such as sensors 104, when a shopper,such as customer 112, enters a check-in area, such as check-in area 110.The unidentified customer data 208 may include multiple sets of data forthe various customers that visit the store.

Unidentified customer data 208 also may include data captured by sensors104 as the customer travels through the store. For example, unidentifiedcustomer data 208 may include images, data that identifies the camerathat captured each of the images, a time stamp for each image, and anidentification value. This unidentified customer data 208 may beanalyzed as disclosed herein when the customer enters the store. Oncethe attributes from the data are extracted it may be linked with thecustomer and the authentication token assigned to the customer uponentering the store.

The unidentified customer data 208 may include an expiration time. As aresult, if the unidentified customer data 208 is not matched withcustomer identity data within a given time, the unidentified customerdata 208 may be flushed from memory 204. The removal of expiredunidentified customer data 208 may allow for faster matching ofunidentified customer data to customer identity data because there areless potential matches. Thus, the risk of mismatching the data is alsoreduced.

Customer identity data 210 may include the authentication token assignedto a customer upon entering the store. The customer identity data mayinclude data supplied by the customer at a kiosk, via a mobile device,etc. as disclosed herein. Customer identity data 210 may be retrievedfrom a database or supplied by the customer. For example, the customermay supply a pointer from his or her mobile device to the server and theserver may retrieve customer identity data 210 from a database using thepointer.

Both unidentified customer data 208 and customer identity data 210 maybe stored as a plurality of feature vectors. The feature vectors may becreated by an encoder that may be part of software module 206. Thefeature vectors may include the various attributes that are extractedfrom data gathered via sensors 104 as disclosed herein. The featurevectors may have the same dimensions or may have different dimensions.For example, feature vectors generated by the encoder for each customerthat enters the store via the check-in area may have differingdimensions depending on the number of attributes the encoder is able toextract from the images captured via cameras 104. Also, as the customerwalks through the store additional attributes may be captured and addedto the feature vector for the customer.

User interface 212 can include any number of devices that allow a userto interface with server 106. Non-limiting examples of user interface212 include a keypad, a microphone, a display (touchscreen orotherwise), etc.

Communications port 214 may allow server 106 to communicate with variousinformation sources and devices, such as, but not limited to, remotecomputing devices, mobile devices, peripheral devices, etc. Non-limitingexamples of communications port 214 include, Ethernet cards (wireless orwired), BLUETOOTH® transmitters and receivers, near-field communicationsmodules, etc. User interface 212 may also use communications port 214 tocommunicate with server 106. For example, user interface 212 may be atable computer that includes a virtual keyboard that communicates withserver 106 via communications port 214.

I/O device 216 may allow server 106 to receive and output information.Non-limiting examples of I/O device 216 include, but are not limited to,a camera (still or video), sensors (such as sensors 104), a weightdetection device such as a scale, a printer for printing receipts andpacking lists, a scanner, etc.

FIG. 3 shows an example method 300 consistent with this disclosure.Method 300 may begin at stage 302 where unidentified customer data maybe received. For example, sensors, such as sensors 104, may captureimages of a customer and transmit the images to a server, such as server106. For instance, as disclosed herein, when customer 112 enters store102 via check-in area 110, sensors 104 may capture various images,weight, etc. of customer 112 and transmit the images to server 106.

From stage 302 method 300 may proceed to stage 304 where customeridentity data may be received. For instance, as disclosed herein, a usermay use a kiosk to enter a username and password, a member number, etc.In another instance, a mobile device of the customer may transmit ausername and password, a member number, etc. to the server.

From stage 304 method 300 may proceed to stage 306 where anauthentication token may be generated. The authentication token may begenerated upon the customer identity data being matched to theunidentified customer data. For example, the authentication token may begenerated by mapping identities to basic authentication strings so thatwhen the customer identifies himself or herself, the customer's basicauthentication string may be returned.

From stage 306 method may proceed to stage 308 where a service may beprovided. As disclosed herein, the service provided may include trackinga customer through a store and tracking items to be purchased upon thecustomer leaving the store. The service may be automatically checkingout after he or she leaves the store. The service could be guiding thecustomer to a particular location. For example, the customer may be atthe store to pick up an item that was previously ordered and paid forand the service may be pushing instructions to the customer's mobiledevice to guide the customer to a pickup area. At the movies, theservice may be authorizing the customer to enter a turnstile and providedirections to an appropriate theater where the movie is showing.

Examples

Example 1 is a method comprising: receiving, at a backend system,unidentified customer data including information about an unidentifiedcustomer; receiving, from the customer, customer identity data, thecustomer identity data including identifying information about thecustomer; and generating an authentication token that links theunidentified customer data to the customer identity data.

In Example 2, the subject matter of Example 1 optionally includeswherein receiving unidentified customer data includes receiving aplurality of images of the customer from multiple cameras.

In Example 3, the subject matter of Example 2 optionally includeswherein the multiple cameras are arranged at different angles relativeto the customer.

In Example 4, the subject matter of any one or more of Examples 2-3optionally include extracting the unidentified customer data from atleast one of the plurality of images.

In Example 5, the subject matter of any one or more of Examples 1-4optionally include wherein receiving unidentified customer data includesreceiving a weight of the unidentified customer.

In Example 6, the subject matter of any one or more of Examples 1-5optionally include wherein receiving the customer identity data includesreceiving the customer identity data from a mobile device of thecustomer.

In Example 7, the subject matter of Example 6 optionally includeswherein the customer identity data is pushed from the mobile device tothe backend system.

In Example 8, the subject matter of any one or more of Examples 1-7optionally include wherein receiving the customer identity data includestransmitting, by the backend system, a request to a mobile device of thecustomer.

In Example 9, the subject matter of any one or more of Examples 1-8optionally include wherein receiving the customer identity data includesreceiving the customer identity data from a kiosk.

In Example 10, the subject matter of any one or more of Examples 1-9optionally include wherein unidentified customer data includes biometricdata of the customer and the customer identity data includescomplimentary biometric data.

Example 11 is a system comprising: a plurality of sensors locatedproximate an entrance to a store; a backend system including aprocessor; and a non-transitory computer-readable storage medium storinginstruction that, when executed by the processor, cause the processor toperform operations comprising: receiving unidentified customer dataincluding information about an unidentified customer, receiving, fromthe customer, customer identity data, the customer identity dataincluding identifying information about the customer; and generating anauthentication token that links the unidentified customer data to thecustomer identity data.

In Example 12, the subject matter of Example 11 optionally includeswherein the plurality of sensors includes a plurality of camerasoperable to transmit a plurality of images of the customer to thebackend system; and receiving unidentified customer data includesreceiving the plurality of images of the customer from multiple cameras.

In Example 13, the subject matter of Example 12 optionally includeswherein the multiple cameras are arranged at different angles relativeto the customer.

In Example 14, the subject matter of any one or more of Examples 12-13optionally include extracting the unidentified customer data from atleast one of the plurality of images.

In Example 15, the subject matter of any one or more of Examples 11-14optionally include wherein the plurality of sensors includes a scalelocated in the floor of the entrance to the store; and receivingunidentified customer data includes receiving a weight of theunidentified customer from the scale.

In Example 16, the subject matter of any one or more of Examples 11-15optionally include wherein receiving the customer identity data includesreceiving the customer identity data from a mobile device of thecustomer.

In Example 17, the subject matter of Example 16 optionally includeswherein the customer identity data is pushed from the mobile device tothe backend system.

In Example 18, the subject matter of any one or more of Examples 11-17optionally include wherein receiving the customer identity data includestransmitting, by the backend system, a request to a mobile device of thecustomer.

In Example 19, the subject matter of any one or more of Examples 11-18optionally include wherein receiving the customer identity data includesreceiving the customer identity data from a kiosk.

In Example 20, the subject matter of any one or more of Examples 11-19optionally include wherein unidentified customer data includes biometricdata of the customer and the customer identity data includescomplimentary biometric data.

It will be readily understood to those skilled in the art that variousother changes in the details, material, and arrangements of the partsand method stages which have been described and illustrated in order toexplain the nature of the inventive subject matter may be made withoutdeparting from the principles and scope of the inventive subject matteras expressed in the subjoined claims.

1. (canceled)
 2. A method comprising: assigning a temporaryidentification value to an unidentified customer; receiving, at abackend system, a query that includes a feature vector comprisingunidentified customer data including the assigned temporaryidentification value and information about the unidentified customer;receiving, from the unidentified customer, customer identity data, thecustomer identity data including identifying information about theunidentified customer; and generating an authentication token that linksthe unidentified customer data and the assigned temporary identificationvalue to the customer identity data, wherein receiving the unidentifiedcustomer data includes receiving a plurality of images of the customerfrom multiple cameras.
 3. The method of claim 2, wherein the multiplecameras are arranged at different angles relative to the unidentifiedcustomer.
 4. The method of claim 2, further comprising extracting theunidentified customer data from at least one of the plurality of images.5. The method of claim 2, wherein receiving the unidentified customerdata further includes receiving a weight of the unidentified customer.6. The method of claim 2, wherein receiving the unidentified customerdata further includes receiving the customer identity data from a mobiledevice of the unidentified customer.
 7. The method of claim 6, whereinthe customer identity data is pushed from the mobile device to thebackend system.
 8. The method of claim 2, wherein receiving theunidentified customer data further includes transmitting, by the backendsystem, a request to a mobile device of the customer.
 9. The method ofclaim 2, wherein receiving the unidentified customer data furtherincludes receiving the customer identity data from a kiosk.
 10. Themethod of claim 2, wherein the unidentified customer data includesbiometric data of the customer and the customer identity data includescomplimentary biometric data.
 11. A system comprising: a backend systemincluding a processor; a plurality of sensors located proximate anentrance to a store, the plurality of sensors including a plurality ofcameras operable to transmit a plurality of images of the customer tothe backend system; and a non-transitory computer-readable storagemedium storing instruction that, when executed by the processor, causethe processor to perform operations comprising: assigning a temporaryidentification value to an unidentified customer; receiving a query thatincludes a feature vector comprising the assigned temporaryidentification value and unidentified customer data includinginformation about the unidentified customer; receiving, from theunidentified customer, customer identity data, the customer identitydata including identifying information about the unidentified customer;and generating an authentication token that links the unidentifiedcustomer data and the assigned temporary identification value to thecustomer identity data, wherein receiving the unidentified customer dataincludes receiving a plurality of images of the unidentified customerfrom the plurality of cameras.
 12. The system of claim 11, wherein themultiple cameras are arranged at different angles relative to theunidentified customer.
 13. The system of claim 11, further comprisingextracting the unidentified customer data from at least one of theplurality of images.
 14. The system of claim 11, wherein the pluralityof sensors further includes a scale located in the floor of the entranceto the store; and receiving the unidentified customer data furtherincludes receiving a weight of the unidentified customer from the scale.15. The system of claim 11, wherein receiving the customer identity datafurther includes receiving the customer identity data from a mobiledevice of the unidentified customer.
 16. The system of claim 15, whereinthe customer identity data is pushed from the mobile device to thebackend system.
 17. The system of claim 11, wherein receiving thecustomer identity data further includes transmitting, by the backendsystem, a request to a mobile device of the unidentified customer. 18.The system of claim 11, wherein receiving the customer identity datafurther includes receiving the customer identity data from a kiosk. 19.The system of claim 11, wherein the unidentified customer data includesbiometric data of the unidentified customer and the customer identitydata includes complimentary biometric data.
 20. A method comprising:receiving, at a backend system, unidentified customer data includinginformation about an unidentified customer from multiple cameras locatedin a store and arranged at different angles relative to the customer;receiving, from the customer at a kiosk, customer identity data, thecustomer identity data including identifying information about theunidentified customer; generating an authentication token that links theunidentified customer data to the customer identity data; and removingthe authentication token, unidentified customer data, and the customeridentity data from memory upon the unidentified customer leaving thestore.
 21. The method of claim 20, wherein receiving the unidentifiedcustomer data includes: assigning a temporary identification value tothe unidentified customer; and receiving a query that includes a featurevector comprising the assigned temporary identification value and theunidentified customer data.